J 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 





@ Publication number: 0 622 931 A2 



EUROPEAN PATENT APPLICATION 



(g) Application number : 94301826.7 
(2) Date of filing ; 15.03.94 



© Priority : 19.03.93 US 34313 

@) Date of publication of application : 
02.11.94 Bulletin 94/44 

(Si) Designated Contracting States : 
DE FR GB 

© Applicant : AT & T GLOBAL INFORMATION 
SOLUTIONS INTERNATIONAL INC. 
1700 South Patterson Boulevard 
Dayton, Ohio 45479 (US) 



© int. ci. 5 : H04L 29/06, H04L 12/18 



@ Inventor : Schwartz, Krista S. 
953 Jennifer Drive 
Batavia, IL 60510 (US) 
Inventor : Carleton, Allison A. 
1425 Leavltt 
Chicago, IL 60622 (US) 
Inventor : FitzPatrick, Catherine M. 
953 Jennifer Drive 
Batavia, IL 60510 (US) 
Inventor : Pommier, Theresa M. 
126 S. Wilmette 
Westmont, IL 60559 (US) 

@ Representative : Robinson, Robert George 
International Patent Department 
NCR Limited 
915 High Road 
North Finchley 
London N12 8QJ (GB) 



(§4) Display annotation for computer collaboration system. 

(5?) In a computer system including a plurality of 
interlinked computers, annotations made on 
the display of one computer are replicated on 
the displays of the other computers. The anno- 
tations may be made in a shared window which 
is a display for an application program running 
on one of the computers. 
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This invention relates to operating a plurality of 
interlinked computers. 

Modern telephone systems allow multiple parties 
at different locations to hold a conference. However, 
telephone conferences do not provide all of the con- 5 
veniences of a face-to-face conference, where par- 
ticipants all meet at a common table in a meeting 
room. For example, in a meeting room, participants 
can view an object of interest, such as a drawing or a 
product. Such viewing is not possible in a telephone 10 
conference. 

It is an object of the invention to provide an im- 
proved electronic conferencing system. 

Therefore, according to the present invention, 
there is provided a method of operating a computer 15 
system, including a plurality of interlinked computers, 
characterized by the steps of: running a message- 
driven operating system on each computer; and when 
any user of a computer generates a message of a pre- 
determined type, replicating the message atthe other 20 
computers. 

One embodiment of the present invention will 
now be described by way of example, with reference 
to the accompanying drawings, in which:- 

Figure 1 illustrates three computers, connected 25 
by telephone lines. 

Figure 2 initiates an example, which will be elabo- 
rated in Figures 3-14. In the exam pie, a calculator pro- 
gram is operated, and annotated, by various parties. 

Figure 3 illustrates how the system responds 30 
when a host user attempts to operate the calculator, 
in Annotation mode. 

Figure 4 illustrates how the system responds 
when a host user attempts to annotate the calculator, 
in Annotation mode. 35 

Figure 5 illustrates how the system responds 
when a host user attempts to operate the calculator, 
in Application mode. 

Figure 6 illustrates how the system responds 
when a host user attempts to operate the calculator, 40 
in Local Annotation mode. 

Figure 7 illustrates how the system responds 
when a host user attempts to annotate the calculator, 
in Local Annotation mode. 

Figure 8 illustrates how the system responds to 45 
a host user when in View mode. 

Figure 9 illustrates how the system responds 
when a remote user attempts to operate the calcula- 
tor, in Annotation mode. 

Figure 10 illustrates how the system responds so 
when a remote user attempts to annotate the calcu- 
lator, in Annotation mode. 

Figure 11 illustrates how the system responds 
when a remote user attempts to operate the calcula- 
tor, in Application mode. 55 

Figure 12 illustrates how the system responds 
when a remote user attempts to operate the calcula- 
tor, in Local Annotation mode. 



Figure 13 illustrates how the system responds 
when a remote user attempts to annotate the calcu- 
lator, in Local Annotation mode. 

Figure 14 illustrates how the system responds to 
a remote user when in View mode. 

Figure 15 and 15A illustrate logic flow used by the 
system. 

Figure 1 shows three computers connected by 
telephone links. Each computer runs a message-driv- 
en, multi-tasking, Graphical User Interface (GUI), 
such as that sold under the name Windows, available 
from Microsoft Corporation, located in Redmond, 
Washington. Such GUIs are also called operating en- 
vironments. 

The user of a GUI interacts with a program byway 
of windows. Selected windows, rather than the entire 
display, are replicated at the remote computers. This 
selective replication allows users to maintain private 
areas on their displays, which are not shared. Each 
computer also runs software which effects the oper- 
ations described herein. In addition, one computer 
(the Host) runs an Application program. (It is possible 
for the Host to run both programs because of the mul- 
ti-tasking capabilities of the GUI.) 

There are four basic modes of operation: 

1. Application Mode 

Any user of any of the three computers in Figure 
1 can issue commands to the Application program. 
For example, assume the Application program is one 
which simulates a hand-held calculator. The initial sit- 
uation is shown in Figure 2, where each computer dis- 
play shows the calculator. Assume that the following 
events occur 

The user of the Host presses the "3" button on the 
calculator (either by keyboard input, or mouse input, 
depending upon the design of the calculator pro- 
gram). 

In response, each calculator, in its display area, 
shows a "3". 

The user of one Remote presses "+ B . 

The user of the other Remote presses "6". 

The user of the Host presses "=". 

At this point, all calculators will display "9", which 
is the sum of 3 and 6. The users collectively operated 
the calculator program, and the display of each shows 
the result. 

The calculator program does not care which 
users pressed the buttons, nor whether some users 
pressed no buttons, provided a legal sequence of but- 
tons was received. (It is assumed that the users are 
cooperative, and that no users try to sabotage oper- 
ation of the calculator.) 

2. Annotation Mode 

Any user can draw on the user's own, local, dis- 
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play, using drawing tools similar to those found in a 
"paint" program. The user can draw boxes, circles, 
arcs, text, ellipses, and so on. The user can also erase 
items on the display. 

The user's annotations are replicated on all other 5 
displays, so that all users view similar displays. How- 
ever, the displays could be different, because of the 
following factors: 

(A) Different display monitors have different 
properties, such as resolution and color capabil- w 
ity. 

(B) Different display protocols (EGA, VGA, etc.) 
represent graphics images differently, and have 
different color capabilities. 

(C) Different GUIs, or different versions of the 15 
same GUI, may have different display conven- 
tions. Different computers in Figure 1 could run 

the different GUIs. 

(D) Some users have changed the size of the win- 
dow in which their calculator is displayed, causing 20 
a deviation in scaling. 

These differences can cause differences in the 
appearance of the displayed images, relative to each 
other, but the basic content of all displays should be 
the same. To accommodate size differences, the dis- 25 
plays may be drawn to different scales as appropriat- 
ed. 

3. Local Annotation Mode 

A user can annotate the local display, but the an- 
notations are kept private, and no other user can see 
the annotations. 

4. View Mode 

No users can annotate, nor can they issue com- 
mands. However, an action resembling annotation 
can be taken. Users can move their cursors, and oth- 
ers will see the movement, allowing remote pointing. 
View Mode is useful in one embodiment, wherein, for 
example, Annotate Mode is in force, but a specific 
user's mode is designated as View. In this embodi- 
ment, all users can annotate, but the "View" user can 
only watch, and cannot annotate. 

Figures 3-14 will illustrate the different modes, 
by way of example, using the calculator program. 

Referring to Figure 3, there is shown the situation 
where the host runs on application program; the mode 
is "Annotation"; the user input is at the host computer; 
and the user attempts to operate the calculator. 

Assume that the user of the host computer at- 
tempts to add two numbers, using the calculator. At- 
tempted entry of the first number will be considered. 

The user, located at the Host, moves the Host's 
cursor over a key of the calculator, as shown in Figure 
3, and tries to depress the key, by clicking the mouse. 
However, the mouse click is blocked and does not 



reach the Application program. The Application pro- 
gram does not respond, because it receives no 
mouse click. 

That is, in more detail, the GUI detects the mouse 
movement, and causes "mouse messages" to be gen- 
erated. The GUi places the mouse messages into a 
queue, where they await processing. INPUT ROUT- 
ER in Figure 1 5 reads these messages. Because "An- 
notation Mode" is currently in force, INPUT ROUTER 
directs the messages to ANNOTATION block. APPLI- 
CATION does not receive the messages and thus 
does not respond. The mouse click is ignored. 

ANNOTATION can be configured to respond to 
two (or more) ways to the mouse messages. In one 
configuration, annotation requires the mouse to ini- 
tially select an ANNOTATION TOOL. If no selection is 
done, ANNOTATION ignores mouse messages. 

Selection is done by clicking the mouse over an 
image of the tool, as is commonly done in "paint" pro- 
grams. ANNOTATION recognizes this tool selection, 
and then treats subsequent mouse clicks as data for 
drawing with the selected tool. For example, if a rec- 
tangle tool were selected, the next two mouse clicks 
would define the diagonal corners of the rectangle. 
(Figure 4, later discussed, illustrates drawing a rec- 
tangle.) 

Under the second configuration, a default tool, 
such as a pen, is automatically selected when in An- 
notation Mode. In this configuration, when the user 
tries to depress a calculator button (by clicking on it), 
the user (unintentionally) initiates drawing of a line, 
using the pen. When the user recognizes this, the 
user can terminate drawing of the line, in any of sev- 
eral known ways. 

Therefore, in Annotation Mode, the system either 
(a) responds to mouse input by initiating a draft an- 
notation, or (b) ignores the mouse input, because an 
annotation tool was not selected. Keyboard input 
from the user is treated the same way. Of course, 
other responses by ANNOTATION can be designed. 

Each display shows a cursor whose position is 
controlled by the associated mouse. Each cursor is 
replicated on all displays. Thus, in Figure 3, with three 
mice, there are three cursors on each display (only 
one is shown for simplicity). Consequently, when one 
user moves a mouse, the corresponding cursor 
moves on all displays. In general, the three cursors 
are distinguishable: each cursor identifies its owner, 
as by color, shape, inclusion of a label, or the like. 

Referring to Figure 4, there is shown the situation 
where the host runs an application program; the 
mode is "Annotation"; the user input is at the host 
computer, and the user attempts to draw a box over 
the calculator. 

This situation is quite similar to that of Figure 3, 
except that, now, the user intends to draw, an annota- 
tion, instead of intending to press a button, as in Fig- 
ure 3. 
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Assume that the user of the host computer draws 
a box over the calculator. (The box is shown overly 
large, for emphasis. It is preferred that the box not ex- 
tend beyond the calculator itself.) The box is replicat- 
ed on the remote computers. (The box is drawn using 5 
annotation tools, which are not shown.) 

In terms of Figure 15, INPUT ROUTER directs 
the logic flow to ANNOTATION. ANNOTATION call 
the proper GDI functions to draw the box. Also, AN- 
NOTATION sends "annotation messages" to CON- 10 
NECTION API, which delivers the annotation mes- 
sages to the Remotes. . 

ANNOTATION in Figure 15A receives the anno- 
tation messages. This ANNOTATION block repre- 
sents the logic executed at each remote computer. 15 
This ANNOTATION calls the proper GDI functions, via 
the block GDI. 

"GDI" is an acronym for Graphical Device Inter- 
face. "GDI functions" are small programs, contained 
in a larger program of the GUI called GDI.EXE. A GDI 20 
function, when called, draws a specific graphic im- 
age, such as a circle, box, or text, based on subse- 
quent input from the user. Other GDI functions per- 
form other tasks, such as selecting pen widths. 

GDI.EXE is a commercially available product. 25 
Technical. details concerning GDI.EXE are contained 
in "Windows Software Development Kit," available 
from Microsoft Corporation, and in Programming 
Windows 3.1 by Charles Petzold (Microsoft Press, 
Redmond, Washington, 1992, ISBN 1-55615-395-3). 30 

Referring to Figure 5, there is shown the situation 
where the host runs an application program; the mode 
is "Application"; the user input is at the host computer; 
and the user attempts to use the calculator. 

The user of the Host moves the cursor over the 35 
calculator key "3" and clicks the mouse. The GUI gen- 
erates a mouse message and places it into the queue. 
The mouse message is read, and passed to the Ap- 
plication program (ie, the calculator program), which 
responds by (1 ) showing that the key "3" is depressed 40 
and (2) drawing the numeral "3" in the calculator's dis- 
play, using GDI calls. The Application program also 
records the fact that the user enters a "3", for its own 
internal operations. 

The GDI calls made by the Application program 45 
in drawing the "3" in the calculator, and in drawing the 
depressed "3" button are intercepted, and notified to 
the other computers. The other computers replicate 
the Host display, by executing the same GDI func- 
tions. Greater detail concerning this GDI interception so 
is given later. 

Thus, all users simultaneously see the user of the 
Host operate the calculator. (The action is not exactly 
simultaneous, because extremely short delays are in- 
volved. However, a human probably could not detect 55 
the delays if the Host and the Remote were operating 
side-by-side.) 

In terms of Figure 15, the INPUT ROUTER rec- 



ognizes that the mouse messages should be directed 
to the Application program, and directs the logic flow 
to APPLICATION (ie, the calculator program). APPLI- 
CATION (1) draws a depressed "3" key and (2) writes 
the numeral "3" on the calculator's display, by calling 
appropriate GDI functions. 

However, the system, via GDI CAPTURE in Fig- 
ure 15, captures the Application program's GDI calls, 
before they are executed. Two things are done with 
the captured calls. One, the other computers are no- 
tified of these calls, via the block CONNECTION API. 
This action leads to block CAPTURED GDI DISPLAY 
in Figure 15A, which causes each Remote to execute 
the same GDI functions, as indicated by block GDI. 
Two, the system allows the GDI functions, called by 
the Application program, to be executed at the host, 
via'the block GDI in Figure 15. Therefore, GDI func- 
tion calls made by the Application Program are cap- 
tured and notified to the Remote computers, so that 
the Remotes can duplicate them. The captured calls 
are then executed as intended on the Host. 

Referring to Figure 6, there is known the situation 
where the host runs an application program; the 
mode is "Local Annotation"; the user input is at the 
host computer and the user attempts to operate the 
calculator. 

Assume that in Annotation Mode, there is no de- 
fault annotation tool given to the user. Under this as- 
sumption, if the user moves the cursor to a calculator 
button, and tries to "press" the button, the INPUT 
ROUTER in Figure 15 passes the mouse message to 
the annotation block. Since the mouse click is not part 
of a valid annotation input sequence (no tool was se- 
lected), ANNOTATION draws nothing. Further, the 
Remote computers do not show the movement of the 
cursor corresponding to the Host computer's mouse, 
as indicated, because line 5 in Figure 15 does not 
send Annotation Messages to the other computers 
when Local Annotation is in force. Further still, the 
calculator button is not redrawn as a depressed but- 
ton on the Host display, in response to the attempt to 
press it, because APPLICATION did not receive the 
mouse message. APPLICATION is responsible for 
drawing depressed calculator buttons. 

If a default annotation is assigned to the user in 
Local Annotation Mode, the user's mouse click would 
initiate drawing by that tool. When the user realized 
the mistake, the user would terminate the drawing, in 
a known manner. 

Referring to Figure 7, there is shown the situation 
where the host runs an application program; the 
mode is "Local Annotation"; the user input is at the 
host computer; and the user attempts to annotate the 
calculator. Under these conditions, the INPUT ROUT- 
ER in Figure 1 5 recognizes a valid attempt to perform 
annotation, as by drawing a box. The INPUT ROUT- 
ER directs the logic flow to the ANNOTATION block, 
which calls the proper GDI functions for drawing the 
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annotation, namely, a box, as shown in Figure 7. How- 
ever, because the annotation is local, no boxes are 
drawn on remote computers, as indicated in Figure 7. 
No data is sent along data path 5 in Figure 15. 

Referring to Figure 8, there Is shown the situation 
where the host runs an application program; the mode 
is 'View"; the user input is at the host computer; and 
the user attempts to operate the calculator. As Figure 
8 indicates, the mouse click is ignored, and nothing 
happens at the Remotes. In Figure 15, the INPUT 
ROUTER reads the mouse message, but blocks it 
from APPLICATION, because the current mode Is 
"view." 

Referring to Figure 9, there is shown the situation 
where the host runs an application program; the mode 
is "ANNOTATION 1 *; the user input is at a remote com- 
puter; and the user attempts to operate the calculator. 

Assume that the user moves the mouse cursor 
over a calculator button and clicks the mouse. The 
mouse click is ignored. The other computers (Host 
and the other Remote) show the motion of the user's 
cursor, but nothing else, because no too! has been se- 
lected. In Figure 15A, the INPUT ROUTER blocks the 
mouse message from reaching APPLICATION. The 
logic is directed to ANNOTATION, which draws a cur- 
sor on the user's Remote display, via block GDI. AN- 
NOTATION also sends data to CONNECTION API, 
which directs the logic to ANNOTATION in Figure 15. 
This ANNOTATION represents the annotation logic 
present on the two other computers: the Host and the 
other Remote. These ANNOTATION blocks draw cur- 
sors corresponding to the users cursor, at corre- 
sponding positions, via the GDI block in Figure 15, 
which represents GDI function calls. The Host can 
use one tool, such as a box-drawing tool, while a Re- 
mote can use a different tool , such as a circle-drawing 
tool. 

Referring to Figure 10, there is shown the situa- 
tion where the host run an application program; the 
mode is "ANNOTATION"; the user input is at a remote 
computer; and the user attempts to annotate the cal- 
culator. Assume that the annotation is a box. A box is 
drawn on all displays. In Figure 15A, the INPUT 
ROUTER at the user's Remote directs the mouse 
messages to the block AN NOTATION. ANNOTATION 
does two things. One, it calls the proper GDI functions 
to perform the annotation, namely, drawing the box. 
Two, ANNOTATION sends annotation messages to 
CONNECTION API, which delivers the annotation 
messages to the other computers. However, one of 
these is the Host, and the other is a Remote. The logic 
at the Host reaches ANNOTATION in Figure 15, and 
the logic at the other Remote reaches ANNOTATION 
in Figure 15A. 

Both of these ANNOTATION blocks cause the 
proper GDI functions to be called, to draw an annota- 
tion corresponding to the user's annotation. However, 
in the Host, logic path 5 is not taken at this time, be- 



cause it is not necessary to replicate the Host's an- 
notations at other computer. 

Referring to Figure 11, there is shown the situa- 
tion where the host runs an application program; the 

5 user input is at a remote computer; and the user at- 
tempts to operate the calculator. It will be recalled that 
the calculator program is loaded only on the host, 
while a Remote user wishes to operate it. 

The Remote user's INPUT ROUTER in Figure 

10 15A routes the mouse messages to CONNECTION 
API. The Host receives these messages, which are 
delivered to the Host's INPUT ROUTER in Figure 15. 
The Host's INPUT ROUTER directs the messages to 
the block APPLICATION (ie, to the Application pro- 
fs gram, namely, the calculator program), which does 
two important things. 

The calculator program treats the messages as 
though they were issued by the Hosfs mouse, even 
though a Remote mouse caused them. The calculator 

20 program responds in its usual way, which includes (1 ) 
showing a depressed calculator button "3", (2) writing 
the numeral "3" in the calculator's display, and (3) 
performing its own internal computations when it 
learns that the user entered data (namely, the "3"). 

25 However, before the calculator program can exe- 

cute (1) and (2) in the previous paragraph, the system 
first captures the GDI functions which the calculator 
program calls. This capture is illustrated in block GDI 
CAPTURE in Figure 15. During this capture, the sys- 

30 tern, in effect, does two things. One, it sends these 
GDI functions to CONNECTION API (for the other 
computers to use). At the user's Remote, CONNEC- 
TION API in Figure 15A directs the GDI functions to 
CAPTURED GDI DISPLAY, which replicates the 

35 Host's display. Two, it causes the GDI functions to be 
executed at the Host (via block GDI in Figure 15). 
Therefore, the general sequence of events is the fol- 
lowing: 

The Remote user attempts to press a calculator 
40 button. 

The software running on the Remote detects this 
attempt, and sends data to the calculator program 
running on the host The data takes the form of mes- 
sages which the calculator program "thinks" come 
45 from the Hosfs mouse. 

The calculator program performs as usual, and 
draws images on the Host display, via GDI calls. 

The system captures the GDI calls, and informs 
the Remotes of them. 
so The Remotes replicate the Host's window. The 
Remote user thus can remotely operate the calculator 
program running on the Host. 

Summarizing in a different way: mouse messag- 
es are generated at the Host, based on mouse mes- 
55 sages at the Remote. The calculator program (run- 
ning on the Host) responds to the mouse messages 
as though they were generated at the Host. The GDI 
calls made by the calculator program are intercepted, 
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and executed at the Remote, thereby replicating the 
Host's display at the Remote. 

Referring to Figure 12, there is shown the situa- 
tion where the host runs an application; the mode is 
"Local Annotation", the user input is at a remote com- 5 
puter, and the user attempts to operate the calculator. 
The user's mouse click is ignored. Nothing appears 
on the other displays in response to the mouse move- 
ment, because of failure to select a tool. 

Referring to Figure 13, there is shown the situa- 10 
tion where the host runs an application program; the 
mode is "Local Annotation"; the user input is at a re- 
mote computer; and the user attempts to annotate the 
calculator. The annotation is drawn on the user's dis- 
play, as indicated. No annotation occurs on the other 15 
displays. 

Referring to Figure 14, there is shown the situa- 
tion where the host runs an application program; the 
mode is "View"; the user input is at a remote comput- 
er; and the user attempts to operate the calculator. As 20 
indicated, the mouse cursor moves at the user's dis- 
play, but the mouse click is ignored. Further, the other 
two displays show the movement of the user's mouse 
cursor. 

With regard to the above situation descriptions, 25 
the following points should be noted. The displays are 
drawn using GDI functions. However, different parts 
of a display are drawn by different programs. 

Despite the fact that all these drawing operations 
are undertaken using GDI functions GDI functions 30 
are not the exclusive medium of communication be- 
tween computers for replicating the displays. For ex- 
ample, when a user performs annotation, the user's 
mouse messages are replicated, as messages, at the 
other computers, via path 5 in Figure 1 5. These repli- 35 
cated messages then cause the respective ANNOTA- 
TION blocks (at the other computers) to issue the 
proper GDI calls for drawing the annotation. That is, 
GDI calls are not send directly from the user perform- 
ing the annotation to the other computers. 40 

in contrast, when an application program causes 
a graphic image to be drawn on a display, GDI calls 
are intercepted via GDI CAPTURE in Figure 15) and 
caused to be replicated on the other computers. 

A major reason for the two different procedures 45 
(replicating mouse messages and replicating GDI 
calls) is that annotations are stored in memory at dif- 
ferent locations than the display information. That is, 
returning to the calculator of Figure 2, the Application 
program stores the image of the calculator in the fol- so 
lowing general way. Annotation data is stored. Appli- 
cation program data is stored by the Application pro- 
gram (at the host). Each image of a key is stored as 
data from which a GDI function can draw the key. The 
data includes information such as position, size, col- 55 
or, and so on. Each key includes an associated num- 
ber. The number can be stored as a text character, 
with information as to position, size, font type, and so 



on. Annotation data is stored at a different location, 
but in the same general way. If either the annotation 
or the Application program needs bitmaps, the bit- 
maps are stored in a conventional, known manner, by 
the GUI. 

The annotation images are combined with the 
Application's images by the known technique of 
masking. That is, at a Remote, the received GDI func- 
tions are executed into a bitmap. The received anno- 
tation information is executed into a different bitmap. 
The two bitmaps are masked together. The annota- 
tion data is kept separate from the application data so 
that, for example, a user can save an Application im- 
age, but without annotations. Alternatively, a user can 
save annotation data alone, or save an annotated dis- 
play. 

As another example, keeping the annotation data 
separate facilitates drawing a display having no anno- 
tation data. If the annotation data were intermingled 
with the calculator image data, elimination of the an- 
notation data would be difficult, if not impossible. If 
GDI calls were transmitted exclusively (ie. no mes- 
sage replication were undertaken), then extra effort 
' would be required to construct annotation data for 
separate storage. 

GDI interception can be understood as follows. 

A. On start-up, the first five bytes of each GDI 
function are replaced with a JUMP instruction to 
a particular program, namely, Trap.GDI. 

B. Trap.GDI gets the parameters for the desired 
graphics image (eg. in the case of a box, the lo- 
cations of the two diagonal corners) and calls the 
sub-program PkgDispCall. Trap.GDI also replac- 
es the first five bytes. 

C. PkgDispCall accepts the parameters from 
Trap.GDI and generates an object structure. This 
object structure is a block of data containing ev- 
erything necessary for the other computers to 
draw the box. 

For example, the object structure contains infor- 
mation as to size and position of the box. Further, the 
GUI draws images within a "context". The context in- 
cludes things such as pen width, color, and other fea- 
tures. The contexts of the individual computers are 
tracked. If the context of the box drawn is different 
from the contexts of the remote computers, PkgDisp- 
Call includes data necessary for the other computers 
to create the correct contexts. 

D. The object structure is shipped to the other 
computers, which then execute the same GDI 
functions. 

E. The invention executed the original GDI func- 
tions. 

The displays are not replicated bit-by-bit. For ex- 
ample, the image of the calculator in Figure 2 could 
be transferred between computers in bitwise fashion. 
If the calculator occupied a space of 200 x 300 pixels, 
then information regarding 60,000 (ie. 200 x 300) pix- 



6 



11 



EP 0 622 931 A2 



12 



eis must be sent. 

Instead, the particular calculator image shown in 
Figure 2 is treated as eighteen rectangles, plus a text 
character for each of sixteen of the rectangles, giving 
a total of 34 objects. Each object requires parameters, 
such as size and position. The number of parameters 
is small, in the range of three to ten. Assuming ten 
parameters, then 340 pieces of data must be sent. Of 
course, the size of each piece depends on many fac- 
tors, but a small number of bytes for each piece may 
be assumed. 

Therefore, the 60,000 pieces of data needed for 
bitwise replication once reduced to 340 pieces maxi- 
mum for object replication. Of course, some objects 
may take the form of bitmaps, and must be sent bit- 
by-bit However, in general, bitmaps are expected to 
be rare. Further, it is expected that, in general, bit- 
maps, when sent, need be sent only once. Also, the 
object data is compressed when possible. That is, ev- 
ery transmission between computers is of com- 
pressed data, when possible. Compression is known 
in the art 

Communication among computers can take sev- 
eral forms. Commercially available networks, local 
and wide area, can be used. Commercially available 
ISDN telephone service, provided by local telephone 
companies, can be used. Modem communication can 
be used. There are commercially available packages 
which detect messages generated by the GUI in re- 
sponse to an input device. One such package is WIN- 
SIGHT, available from Borland International. 

An alternative approach to the graphics capture 
described above is the following. The system- provid- 
ed GDI is replaced by a separate procedure which 
processes GDI calls before calling the actual system 
GDI. The system GDI name is changed to prevent 
confusion between the two modules. The same tech- 
nique is also used on USR.EXE to also capture GDI 
calls made through system-provided modules. 

A given computer can act as a Host for one pro- 
gram and a Remote for another. For example, one 
computer can run a word processing program. An- 
other computer can run a CAD drawing program. 
Each is Host for its respective program. 

Since the software on each computer is identical, 
or substantially identical, all users can run either the 
word processing program or the CAD program, in the 
manner described above. 

There are two types of "cursor". Each GUI gen- 
erates its own "real" cursor. The real cursor is not gen- 
erated by GDI functions, but by an independentfunc- 
tion in the GUI. The reader can view the cursor as a 
bitmap which the GUI moves in response to mouse 
motion. 

In addition to the real cursor, which is controlled 
by the local mouse, the system generates a "pseudo" 
cursor for each remote participant. The pseudo cur- 
sors are generated using GDI functions. 



Sometimes a real cursor changes shape as the 
cursor moves. For example, it can take the form of an 
arrow when lying on a tool bar, and then change to a 
hand when lying on a client area. Sometimes this 

5 change is under the control of the Application pro- 
gram. Therefore, if Remote user is controlling an Ap- 
plication program running on a Host machine (as in 
Figure 11), the Application program may change the 
cursor on the Host machine, but without using GDI 

10 calls. Consequently, the GDI capture of Figures 15 
and 1 5A will be ineffective to replicate the changes on 
the Remote display. 

To confront this problem, the system watches for 
the functions which change the real cursor (eg. the 

15 SetCursor command). The cursor change is replicat- 
ed on the Remote computer. One way is to execute 
the same SetCursor command. An alternative ap- 
proach would be to change the Remote cursor by exe- 
cuting a proper sequence of GDI calls, or to draw a bit- 

20 map, when the Host cursor changes. 

It should be understood that the system only rep- 
licates windows which the user of a display identifies. 
That is, the user can keep a workspace, such as a 
notepad, private during a conference. GDI calls use a 

25 task handle. If the task handle does not refer to a 
shared item, the GDI calls are not shared. 



Claims 

30 

1. A method of operating a computer system, includ- 
ing a plurality of interlinked computers, charac- 
terized by the steps of: running a message-driven 

' operating system on each computer; and when 

35 any user of a computer generates a message of 

a predetermined type, replicating the message at 
the other computers. 

2. A method according to claim 1 , characterized by 
40 the steps of capturing parameters indicative of 

the location of a cursor image at one of said com- 
puters; sending the parameter to the other com- 
puters; and generating a cursor image on the 
other computers, based on the parameters. 

45 

3. A method of operating a computer system ac- 
cording to any one of the preceding claims, char- 
acterized by the steps of drawing a graphic image 
on a display of one of the computers, in response 

so to messages; transmitting similar messages to 

other computers; and causing said graphic image 
to be displayed at the other computers. 

4. A method according to any one of the preceding 
55 claims, characterized by the steps of displaying 

visual data in windows of the displays of said 
computers; replicating at least one window in all 
of said displays; and allowing a user to prevent 



f 
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replication of a selected window on the user's dis- 
play. 

5. A method according to any one of the preceding 
claims, characterized by the steps of running an 5 
application program on one of said computers 
acting as a host computer and allowing a selected 
remote computer to issue commands to said ap- 
plication program. 

10 

6. A computer system for carrying out a method ac- 
cording to any one of the preceding claims, in- 
cluding a plurality of interlinked computers, and 
control means adapted, when any user of a com- 
puter generates a message of a predetermined is 
type, to replicate the message at the other com- 
puters. 



25 



30 



35 



40 



45 



50 



55 



BP 0 622 931 A2 




EP 0 622 931 A2 





10 



EP 0 622 931 A2 



2T 

g 

CO < 
HH < 

o 





LU 
\— 

O 

LU 

DC 



< 

DC 

OLU v 

. *T CO 
<LLJ7< 



O 1 



< 

o 



CO 
LU 



SO co 
h- 

coo 

§2 

>co 

o ^ 
oo 



DC 



LU 

□c 

LU 





CO CO 
l-UJ 
CO> 

OO 
xs 

Oo 
>co 

Q- DC 

oo 




CO 

o 



LU 
DC 
LU 
X 

a 



3 
a: 

CO 



< 

DC 
O 

o 

DC 
CL 



LU 
I— 

o 



11 



EP 0 622 931 A2 




12 



r 




13 



EP 0 622 931 A2 




14 



BP 0 622 931 A2 




15 



EP 0 622 931 A2 




16 



EP 0 622 931 A2 




17 




18 



EP 0 622 931 A2 




BP 0 622 931 A2 




20 



EP 0 622 931 A2 




EP 0 622 931 A2 




23 



(19) 



J) 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 



(12) 



(88) Date of publication A3: 

29.05.1996 Bulletin 1996/22 



mm 

(n) EP 0 622 931 A3 

EUROPEAN PATENT APPLICATION 

(51) mtci » H04L 29/06, H04L 12/18 



(43) Date of publication A2: 

02.11.1994 Bulletin 1994/44 

(21) Application number: 94301826.7 

(22) Date of filing: 15.03.1994 



(84) Designated Contracting States: 


• Carleton, Allison A. 


DE FR GB 


Chicago, IL 60622 (US) 




• FitzPatrick, Catherine M. 


(30) Priority: 19.03.1993 US 34313 


Batavia, IL 60510 (US) 


• Pommier, Theresa M. 


(71) Applicant: AT&T GLOBAL INFORMATION 


Westmont, IL 60559 (US) 


SOLUTIONS INTERNATIONAL INC. 




Dayton, Ohio 45479 (US) 


(74) Representative: Robinson, Robert George 


International Patent Department 


(72) Inventors: 


NCR Limited 


• Schwartz, Krista S. 


915 High Road 


Batavia, IL 60510 (US) 


North Finch ley 




London N12 8QJ (GB) 



(54) Display annotation for computer collaboration system 



CO 
< 
l— 

CO 

o> 

CM 
CM 
CO 

o 

Q. 

LU 



(57) In a computer system including a plurality of in- 
terlinked computers, annotations made on the display 
of one computer are replicated on the displays of the 
other computers. The annotations may be made in a 
shared window which is a display for an application pro- 
gram running on one of the computers. 
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